Stimulus Integration Apparatus

ABSTRACT

Apparatus, architecture and methods for integrating and/or presenting stimuli to sensory receptor(s) are provided. An electronically stored value may encode stimuli presentable via electronic display(s) to receptor(s). Stimulus/stimuli may include “attended” and/or “unattended” data. Data may mediate neurophysiological benefit(s). Data may include visual, motion and/or auditory information. Benefits) may include motion sickness alleviation. “Unattended” data may include translucent moving patterns and/or shapes and/or shifts in screen luminance, such as flashing. “Unattended” data may be associated with detected motion function(s) to convey sense of motion. “Unattended” data may be configured to be presented so as to avoid being distracting. “Unattended” data may be configured to be presented in a periphory(ies) of user(s). Display(s) may be handheld, worn and/or vehicle-mounted.

This application is a continuation of U.S. patent application Ser. No.14/799,350, filed Jul. 14, 2015, which is a divisional of U.S. patentapplication Ser. No. 13/207,964, filed Aug. 11, 2011, now U.S. Pat. No.9,123,143. U.S. application Ser. Nos. 13/207,964 and 14/799,350 areincorporated by reference herein in their respective entireties.

FIELD OF THE INVENTION

The present disclosure relates generally to a system and method forsubstantially simultaneously presenting different sets of displayabledata on the same display screen for minimizing motion sickness. Inparticular, the present disclosure relates to a system and method forsubstantially simultaneously displaying a set of attended data that auser wants to attend to, and a set of unattended data on the samedisplay screen for minimizing motion sickness, wherein the unattendeddata and attended data are combined without obstructing or interferingwith one another.

BACKGROUND

The brain is hardwired for processing parallel sensory data-streams inan automatic manner. Some of these sensory processes are consciouslylearned and reliant on attention, such as browsing the web or watching amovie. Others are subconscious and instinctive, such as the detection ofneurotoxin influence in the brain, bused on discordant multisensoryevents. In the modern world, some of these instinctive processes areless than beneficial. For instance, the neural system for detectingneurotoxins triggers motion sickness in otherwise healthy individualswhen they travel in moving vehicles. In fact, motion sickness is anailment that afflicts countless millions and inhibits their ability toperform basic tasks, such as reading while traveling.

Motion sickness is due to discordant multisensory events that may occurwhen a person is travelling while attending visually to something otherthan the view that is ahead of the person. The vestibular system sensesthe motion, whereas the visual system acquires information that isinconsistent with that acquired by the vestibular system. Theinconsistency or multisensory discord may trigger a defensive biologicalresponse hardwired for the detection and elimination of ingestedneurotoxins that lead to similar sensory-discordance in the mammalianbrain.

SUMMARY

The present disclosure is directed to a system for substantiallysimultaneous data presentation. The system includes at least onetangible processor and a memory with instructions to be executed by theat least one tangible processor for receiving attended data, includingdata for a user to attend to by paying attention to the data; receivingunattended data, which includes data for the user to sense and processeven when the user does not attend to it; receiving motion data, whichcorresponds to motion that is sensed by a sensing device or simulatedmotion; modifying the unattended data in accordance with the receivedmotion data; and generating presentation data. The generatedpresentation data is adapted and configured for presentation by apresentation device for substantially simultaneous presentation of theattended data and the modified unattended data in a manner in which theunattended data at least partially overlaps the attended data and theunattended data does not interfere with the attended data by replacing,erasing, or suppressing the attended data, and vice versa. The presentedmodified unattended data gives cues about the sensed or simulated motionfor the user to sense and process for compensating for discordantmultisensory information and minimizing motion sickness while attendingto the presented attended data, without a need for the user to attend tothe presented unattended data.

The present disclosure is also directed to a system for substantiallysimultaneous data presentation. The system includes at least onetangible processor and a memory with instructions to be executed by theat least one tangible processor for: receiving attended data, includingdata for a user to attend to by paying attention to the data; receivingunattended data, which includes data for the user to sense and processeven when the user does not attend to it; and generating presentationdata. The generated presentation is adapted and configured forpresentation by a presentation device for substantially simultaneouspresentation of die attended data and the unattended data in a manner inwhich the unattended data at least partially overlaps the attended dataand the unattended data does not interfere with the attended data byreplacing, erasing, or suppressing the attended data, and vice versa.The presented unattended data gives unconscious cues for the user tosense and process for providing information to the user while attendingto the presented attended data, without a need for the user to attend tothe presented unattended data.

The present disclosure is further directed to a method for substantiallysimultaneous data presentation. The method includes receiving by atleast one hardware processor attended data that is suitable fordisplaying on a display device, including data for a user to attend toby paying attention to the data. The method further includes receivingby the at least one hardware processor unattended data suitable fordisplaying on the display device, which includes data for the user tosense and process even when the user does not attend to it, andreceiving by the at least one hardware processor motion data, whichcorresponds to motion that is sensed by a sensing device or simulatedmotion. The method also includes modifying by the at least one hardwareprocessor the unattended data in accordance with the received motiondata, and generating by the hardware processor presentation data. Thegenerated presentation data is suitable for displaying on the displaydevice and provides for substantially simultaneous display of theattended data and the modified unattended data in a manner in which theunattended data at least partially overlaps the attended data and theunattended data does not interfere with the attended data by replacing,erasing, or suppressing the attended data, and vice versa. The displayedmodified unattended data gives cues about the sensed or simulated motionfor the user to sense and process for compensating for discordantmultisensory information and minimizing motion sickness while attendingto the displayed attended data, without a need for the user to attend tothe displayed unattended data.

The present disclosure is yet further directed to a data integrationsystem including at least one tangible processor and a memory withinstructions to be executed by the at least one tangible processor forreceiving two sets of audio data from respective independent soundsources, receiving a first and second control signal, selectivelymodulating either of the received audio data sets in accordance with thefirst control signal, combining the two audio data sets into a combinedaudio data set in accordance with the second control signal, andoutputting the combined audio data set to a single output device. Thecombined audio data set is adapted and configured for a user to attendto an audio data set selected from the two audio data sets, wherein theselection is via the second control signal.

Other features of the presently disclosed system for substantiallysimultaneous data presentation will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, which illustrate, by way of example, the presently disclosedsystem.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present disclosure will be described belowwith reference to the figures, wherein:

FIG. 1 is block diagram of an exemplary motion sickness minimizationsystem in accordance with the present disclosure;

FIG. 2 is block diagram of an exemplary hardware processor incommunication with a data storage device in accordance with the presentdisclosure;

FIG. 3A shows an exemplary unattended pattern in accordance with thepresent disclosure;

FIG. 3B shows exemplary complementary patterns that when displayed in anoscillating fashion produce an illusion of movement;

FIG. 4 shows an example of a screen shot of combined visual attendeddata and modulated unattended data patterns at a first time t.sub.1;

FIG. 5 shows an example of a screen shot of combined visual attendeddata and modulated unattended data patterns at a second time t.sub.2subsequent to t.sub.1;

FIG. 6 shows exemplary first and second screen shots of combined visualattended data and modulated unattended data patterns taken at subsequenttimes in which the unattended data pattern uses a logo;

FIG. 7 is a schematic diagram of an exemplary network based system foradvertising by distributing unattended data;

FIG. 8 shows an audio signal integration system in accordance with anembodiment of the present disclosure; and

FIG. 9 shows a schematic diagram of the audio signal integration systemshown in FIG. 8.

DETAILED DESCRIPTION

Referring now to the drawing figures, in which like reference numeralsidentify identical or corresponding elements, the motion sicknessminimization system and method in accordance with the present disclosurewill now be described in detail. With initial reference to FIG. 1, anexemplary motion sickness minimization system in accordance with thepresent disclosure is illustrated and is designated generally as motionsickness minimization (MSM) system 100.

MSM system 100 includes a hardware processor 102 receiving attendeddata, unattended data, and motion data. The motion data corresponds tomotion that is sensed by a sensing device. The hardware processor 102processes the attended data and unattended data so that they arepresented by a data presentation device 104 substantially simultaneouslysuch that the unattended data does not obstruct or interfere with theattended data, e.g., by replacing, erasing, or suppressing the attendeddata, and vice versa. This substantially simultaneous presentation ofthe attended and unattended data may occur so that the unattended dataat least partially overlaps the attended data and, e.g., displayed atthe same pixel location or adjacent or very close pixel locations thatwhen displayed appear as if they are displayed in the same location. Toaccomplish this, the processor 102 combines the attended and unattendeddata so that they are both presented to the user.

When motion data is received, the processor 102 modifies the unattendeddata in accordance with the received motion data. The modifiedunattended data, which is presented to the user via the datapresentation device 104, gives cues about the sensed motion. Themodified unattended data is combined with the attended data so that theyare presented substantially simultaneously by the data presentationdevice 104. Additionally or alternatively, the unattended data may bemodified in accordance with other types of input, including input from asensor or user input.

The user can attend to the attended data while receiving motion cues viathe modified unattended data. Attending to data refers to payingattention to the data on a conscious level, such as when the user ispaying attention to the data that the user senses and purposefullyprocessing it, for example, reading or mentally absorbing content ofsensed data. The user senses the unattended data but may not beattending to the unattended data. Unattended data is data that the usersenses and which the user's brain processes where the sensing andprocessing can be on a subconscious level. The user does not need to payattention to the data or purposefully process it. The user may beunaware of the data contents. The user may be absorbed in processing thecontents of the attended data and pay no attention to the unattendeddata. The motion cues are modifications to the unattended data based onreceived motion data. The user can sense and process the motion cueswithout attending to the unattended data. As the user attends to theattended data, the user's brain uses the motion cues to compensate fordiscordant multisensory information to reduce motion sickness, includingavoiding the onset of, alleviating, and/or treating motion sickness,without a need for the user to attend to the unattended data.

Hardware processor 102 can be a processor included with a host device,such as a commercially available device, e.g., a personal computer; acomputerized system incorporated into a moving vehicle or gadget, suchas a boat, car, plane, or a virtual reality wearable device (e.g., aheadset); a computerized system incorporated into a stationary systemthat simulates motion; a laptop computer; a television screen; ahandheld computing device, such as a smartphone, a personal digitalassistant (PDA), or a cellular phone; or a system that uses threedimensional (3D) display technology (i.e., capable of conveying astereoscopic perception of 3D depth to the viewer, where the user may ormay not use 3D glasses), etc. Processor 102 may include one or moreprocessors.

As shown in FIG. 2, the processor 102 communicates with at least onedata storage device 202 that may be external or internal to the hostdevice. The at least one data storage device 202, which may includevolatile and nonvolatile memory, stores various data, such asprogramming instructions (e.g., an operating system, one or moreapplication programs, a network (e.g., Internet) browser, etc.) forexecution by the hardware processor 102, data received from sensors thatare internal or external to the host device, and/or application datainput to or output from software applications executed by the hardwareprocessor 102 or other processing devices. Additionally, the datastorage device 202 stores a database of unattended basic patterns 204storing at least one unattended basic pattern which is input to theprocessor 102 for the processor to modulate in accordance with detectedmotion.

A motion function software module 206, a data combination softwaremodule 208, and an attended data processing module 210 are provided forexecution by the processor 102, which each include a series ofprogrammable instructions capable of being executed by the processor102. The series of programmable instructions can be stored on acomputer-readable medium accessible by the processor 102, such as RAM, ahard drive, pen drive, DVD, Blu-ray Disk, CD, Smart Card, 3.5″ diskette,SD card, micro SD card, or any other data storage device, for performingthe functions disclosed herein and to achieve a technical effect inaccordance with the disclosure. The motion function module 206, datacombination module 208, and attended data processing module 210 providefunctional and palpable applications in the field of computertechnology. The functions of the motion function module 206, datacombination module 208, and attended data processing module 210 may becombined into one module or distributed among a different combination ofmodules. The software modules 206, 208, and 210 may be downloadable,such as by a smartphone, a smart consumer electrical device, or apersonal computer, from a remote source, such as a website (e.g., an Appstore that sells applications for smartphones).

With reference to FIGS. 1 and 2, the data presentation device 104presents data output by the data combination module 208 so that thepresented data can be sensed and processed by the user. The datapresentation device 104 may include, for example, an electronic displaydevice for visually displaying digital or analog data, such as acomputer screen, an LCD screen, a digital reader, a television screen, agaming device screen, a display screen for a cellular phone or asmartphone, a projector, a display screen for displaying 3D technology(with or without glasses), etc; an audio output device, such as speakersor headphones; or an output device that outputs a physical energy orforce that is sensed by the somatosensory system, such as an air-blowingdevice, e.g., a fan that outputs blowing air, a vibrator that outputstactilely sensed vibrations, a radiator that outputs heat (e.g., poweredby an existing heat source in the moving vehicle, such as car engine orheater), an air conditioner or refrigerator system that outputs coldair, or a compressor, vent, or valve that can adjusts the pressure,e.g., in a passenger compartment of a moving vehicle. The MSM system 100may use two or more presentation devices 104 at a time so that modulatedunattended data may be presented to the user substantiallysimultaneously via the two or more presentation devices 104.

The attended data is data provided to the processor 102 that the userwants to attend to. Attending to the data includes paying attention tothe data on a conscious level. For example, the attended data mayinclude data, such as text, graphics, or video that is displayed on adisplay device included with the data presentation device 104. Thedisplayed attended data may be an electronic book or journal; aword-processing, spreadsheet, or graphics document; communication-basedtext (email, SMS messages, chats); presentation data (e.g., PowerPoint™); photographs; a movie; gaming visual data; simulation visualdata; or virtual reality visual data, web browser, website, video, etc.The attended data may include audio data, such as a soundtrack to amovie, music, gaming sound data, etc. The attended data may includecontrol data for controlling a mechanical force that is tactilelypresented to the user, such as for controlling an air-blowing device, avibrating device, a pressure control device, or a temperature controldevice. Attending to the tactilely presented data is includes thesomatosensory system sensing the tactile input as a primary input.

The attended data processing module 210 includes software and/orhardware for processing the attended data. The attended data processingmodule 210 may process or execute, for example, an operating system, anapplication software program, a web browser program, etc. Programsprocessed by attended data processing module 210 may be stored, forexample, by the data storage device 202.

The unattended data is provided to the processor 102 from the databaseof unattended basic patterns 204 stored by storage device 202 orreceived as streamed data. The unattended data may include data suitablefor video or audio presentation or for control of a mechanical forcethat can be presented to the user, but that div user dues not have tomend to in order to sense and process it. Accordingly, the unattendeddata may include a graphical screen or a series of graphical screens,e.g., a video data stream, for display; an audio file or streamed audiofile for audio presentation, such as music or noise (e.g., white, pink,brown, popcorn, or random); and/or a control force or series of controlforces for controlling a mechanical force for tactile presentation tothe user. The unattended data may be received by the motion functionmodule 206 where it is modulated based on an input, such as sensedmotion, after which it is combined with the attended data by the datastream combination module 208, as described further below.Alternatively, the unattended data may be combined with the attendeddata by the data stream combination module 208, as described furtherbelow, without modulation by the motion function module 206.

The unattended data may be processed by the unattended data processingmodule 212 before it is provided to the motion function module 206 orthe data stream combination module 208. The unattended data processingmodule 212 may process an unattended data pattern, which may include asingle pattern or a continuous pattern, which may include, for example,a repeating pattern or non-repeating pattern that is continuallyprovided to the motion function module 206 or the data steam combinationmodule 208. The repeating pattern may be output by the unattended dataprocessing module 212, for example, as a data stream. The unattendeddata processing module 212 may also modify the pattern in accordancewith a pattern modification routine, which may be predefined, usergenerated, or random. A plurality of pattern modification routines maybe stored with the unattended basic patterns 204. The patternmodification routine and/or related parameters may be user selectable.The unattended data processing module 212 may include a graphical userinterface (GUI) through which a user may communicate with the unattendeddata processing module 212, such as for selecting or generating anunattended basic pattern, or for generating or selecting a patternmodification routine or modifying parameters to a pattern modificationroutine.

Unattended data may be presented to the user substantiallysimultaneously via different presentation devices 104 that use differentmediums. Unattended data may include one or more patterns selected fromthe database of unattended basic patterns 204, and may includeindependent objects that can be modified independent of the rest of thepattern. Modulation can be applied to a pattern as a whole or toindividual objects. One or more objects may be related to one other,wherein they may be displayed as discrete graphical elements, but asingle modification may affect all of the related objects.

When presented, the unattended data may be sensed and processed by theuser, but the user need not attend to the data and it can be sensed andprocessed on a subconscious level while the user is consciouslyattending to the attended data. With respect to tactilely presentedunattended data, the data sensed by the somatosensoty system as asecondary input that combines with but does not override the attendeddata. The presentation of the unattended data does not interfere withthe user having the ability to attend to the attended data. The twodata-streams are combined and substantially simultaneously presented tothe user in a manner that is both effective and not overly distracting.

The database of unattended basic patterns 204 stores at least onepattern that is suitable for presentation via the presentation devices104. A visual pattern may include, for example a displayable pattern ofshapes, such as dots, wavy lines or bands, geometric shapes, or a logo,displayed in gray scale or color. The visual patterns are conducive toadjustments that indicate motion. Adjusting the position on the displayof such shapes, groups of shapes, or associated edges (oftennon-symmetric), can demonstrate motion that corresponds to vectorsassociated with sensed motion, i.e. horizontal, vertical, orthogonal,and rotational. Corresponding adjustments to the shapes, groups ofshapes or associated edges may include shifting in a horizontaldirection, shifting in a vertical direction, dilating, and rotating.

In an embodiment, the visual pattern includes a static image or a seriesof at least two static images that are displayed consecutively andrepetitively for creating an optical illusion of motion. When theunattended data forms an optical illusion that provides the user with asense of motion, it may be displayed in combination with attended datawithout modulation by motion data and still provide relief from motionsickness. One example of a single static image that produces a sense ofmotion is a pattern well-known as “Rotating Snakes,” Association forResearch in Vision and Ophthalmology (2009, February 3). Not Just YourImagination: Brain Perceives Optical. In another example, the visualpattern includes a series of a first and second static image, whereineach image includes a series of concentric darker and lighter bands thatcan create the illusion of motion through a tunnel. The second imagediffers relative to the first image, such as by reversing which bandsare lighter and which are darker, or by slightly offsetting the positionof the bands in a selected direction. The visual pattern is displayed sothat the presentation oscillates between the first and second images. Inanother example, a strobe light effect can be used to mimic the effectof motion on the visual system.

FIG. 3A shows an exemplary visually displayable unattended basic pattern302. Pattern 302 may be adapted and configured to include individualobjects or as a unitary pattern without objects FIG. 3B shows unattendeddata having a series of patterns including a first pattern 320 and asecond pattern 322 that complements pattern 320. A viewer may experiencea sense of motion when the unattended data is displayed by oscillatingbetween the first and second patterns 320 and 322 (i.e., displayingfirst pattern 320 followed by displaying second pattern 322,repetitively). As each of the first and second patterns 320 and 322 isdisplayed it is combined with attended data including text 330 and taskbars 332 so that the unattended data and data are displayedsubstantially simultaneously. The rate of oscillation may be userselected or be set to a default rate. Due to the complementary nature ofthe first and second patterns 320 and 322 and the nature of the patterns320 and 322 (here, concentric ovals that create a tunnel illusion), anoptical illusion is created by oscillating between the patterns 320 and322. The brain interprets oscillation between the first and secondpatterns 320 and 322 as deceleration or acceleration, producing a senseof motion, which may help to create a balance between vestibularsensations of actual motion and visual cues. Since a sense of motion isproduced, this type of series of patterns may be used to prevent ortreat motion sickness with or without sensing actual motion or adjustingthe unattended data in accordance with sensed motion.

An audio pattern may include for example, an audio file of voice, music,or sound (e.g., white, brown, pink sound, etc.). A control pattern mayinclude a control signal or a series of control signals, e.g., having asteady, pulsed, or wave pattern.

The motion data is provided to the processor 102 by a sensing device(not shown) that senses motion, such as rotation and linear and angularvelocity, acceleration, jerk, and gravity. The sensing device may sensemotion in 3-dimensions and output data that represents the sensed motionin 3-dimensions. The sensing of motion mimics vestibular detection ofmotion. Sensing devices for sensing motion include, for example, anaccelerometer, global positioning system (GPS), gyroscope, or mechanicalsensor. Alternatively, when the motion is simulated by a simulator (notshown), the motion information is provided to the processor 102 by thesimulator. When the motion is simulated, such as by a computing system,such as a simulator or virtual reality game or training system, themotion data is based on the simulated motion generated by the computingsystem.

The motion function module 206 modulates the received unattended data inaccordance with the motion data, e.g., modifying one or more valuesassociated with characteristics of the unattended data as a function ofa characteristic of motion, such as motion in the x, y, and/or zdirection, acceleration (the 1.sup.st derivative of motion), jerks (the2.sup.nd derivative of motion). Alternatively, modulation of theunattended data could be in accordance with rotational or tilting, suchas by calculating expected detection by the vestibular system'ssemicircular canals (which corresponds to sensing rotational movement)and otolith organs (which corresponds to sensing a person's head tilts).

Modulation of a visual unattended basic pattern may include a shift inone or more selected characteristics of the pattern, such as contrast,resolution, chromaticity, brightness, luminance, hue, saturation, color,color balance, size and/or location. Shifts may affect the entirepattern or one or more objects included in the pattern. Location shiftsmay be, for example, horizontal, vertical, a combination thereof, orrotational. For example, a location shift to the right may correspond toa sensor-detected movement to the left. The sensor detection of motionfurther corresponds to the vestibular system's detection of movement.

In a 3D setting, e.g. where the presentation device includes a displayscreen adapted and configured for displaying in 3D, with or without auser using 3D glasses or a head mounted device, binocular detection of3D motion can also be manipulated by representing perspectives of a 3Dunattended pattern that correspond to the disparate images sensed byeach eye's perspective.

The display device may show motion in 3-dimensions (e.g., represented bythree perpendicular axes), such as by using a pattern that isgraphically depicted in 3-dimensions, e.g., showing depth. The motionfunction module 206 may modulate the unattended data so that detectedmovement in each dimension of a 3-dimensional space the user is locatedin corresponds to a location shift of the pattern or a pattern object ina corresponding graphically depicted dimension, including simulateddepth. The user may be able to select a technique for displaying the3-dimensional motion data on the 2-dimensional display screen. Forexample, the user may select which axes in the 3-dimensional space thatthe motion takes place in (i.e., the user's space) corresponds to therespective axes in to the 2- or 3-dimensional display that is displayedon the 2-dimensional display screen. Alternatively or additionally, themotion function module 206 may use two or more different types of shiftsto correspond to motion detected in respective dimensions of the user's3-dimensional space.

In one example, the motion function module 206 may modulate a patterndisplayed on a 2-dimensional display screen of the display device inaccordance with sensed accelerations that are orthogonal to the2-dimensional display screen by adjusting the size of the pattern orpattern objects. Such accelerations, for example, may correspond toupward or downward motion which is orthogonal to the display screen thatis oriented parallel to the ground. This may be the case for anelectronic reader held on a lap or a screen located on a table or deskconsole. For example, such motion may correspond to the up and downmotion of a car bumping along a rough road or boat rocking on waves.Similarly, such accelerations may correspond to accelerations directedin a forward or backward direction which are orthogonal to the displayscreen when it is oriented in a relatively upright manner, e.g. as infor the display screen of a laptop computer or an in-vehicle movieplayer. The motion function module 206 may modulate the pattern orpattern objects based on sensed accelerations or jerks (e.g., the firstderivative of acceleration) orthogonal to the display screen byadjusting the size of the pattern or object, e.g., by shrinking of orenlarging the pattern or pattern object.

The modulations may be smooth or pulsed. For example, the luminance maybe shifted in a stroboscopic manner. Stroboscopic shifts may beparticularly effective, as stroboscopic modulations have recently beenshown to reduce motion sickness in astronauts. Parts or regions of animage (e.g. center and/or periphery) or individual objects in an imagemay be modulated individually or as groups of objects, or the entireimage limy be modulated. Other visual modulations are envisioned.

Modulation of audio unattended data may include varying bandwidth,frequency, center frequency, amplitude, pitch, intervals betweendiscrete sounds. Other modulations are envisioned. Modulation of controlforces may include, for example, varying the direction of the force, theamplitude, the frequency (e.g., for vibrations), etc. For example, whencontrolling a fan, the fan may be controlled to blow air towards theuser so that as the unattended data is modulated it mimics wind that theuser would feel if the user were exposed to wind while experiencing thesensed motion, inducing a somatosensory sensation of motion.

In addition to modulating the unattended pattern, the motion functionmodule 206 may modulate attended data or objects within the attendeddata. For example, text that the user is attending to may be shiftedbriefly in response to sensed accelerations

FIGS. 4 and 5 show screen shots 400 and 500, respectively, in which thevisual unattended pattern 302 is displayed as a luminance modulation ofthe attended data 402, and further displaced in accordance with receivedmotion data. The screen of attended data 402 includes a word processingwindow that includes display of a textual document 404 and task orfunctional areas 406 on the periphery that provide the user with accessto application and operating system functions. The screen of attendeddata further includes a task bar at the bottom that provided access tooperating system functions. Screen shots 400 and 500 were taken at timest.sub.1 and t.sub.2 (where t.sub.2 is chronologically after t.sub.1),respectively, as motion data was received and processed by motionfunction module 206 to modulate the unattended pattern 302. The locationof pattern 302 shown in FIG. 5 is modulated via a vertical (downward)and horizontal (toward the right) shift relative to FIG. 4. Themodulation is in response to received motion data that describing motionin the user's space that corresponds to the upward and left shifts suchas are sensed by the vestibular system.

The data combination module 203 combines the attended data and themodulated unattended data and outputs the combined data to the datapresentation device 104. The two sets of data, the attended data and theunattended data, are included in the combination so that one data setdoes not interfere (e.g., by replacing data or causing data to be erasedor suppressed) with data in the other data set. In one embodiment, eachdata set is formed of elements and corresponding elements of the twosets of data are combined for generating a corresponding element in thecombined data. The unattended data is combined with the attended data sothat it overlaps the attended data so that it has an area or range incommon with the attended data. This may include interleaving elements ofthe unattended data with elements of the attended data or combiningvalues associated with elements of the unattended data and elements ofthe attended data.

For example, when the data sets include visually displayable data, theelements may be image elements of an array of image elements (e.g., abit-map) that are presented via an array of pixels on a display device.Corresponding image elements may be associated with the same (x,y)coordinates, location, or pixel on the screen of the display device thatthey will be displayed on. Corresponding image elements, including afirst pixel from the attended data set and a second pixel from theunattended data set having the same coordinates (x,y), are combined,such as by performing a mathematical function on values associated withone or more properties (e.g., chromaticity, hue, luminance, brightness,intensity, contrast, and/or color) having the same coordinates (x,y)) ofboth of the corresponding pixels. The mathematical function may includeaddition, subtraction, multiplication, division, inversion, powers,etc., or a combination thereof. Another type of combining includesinterleaving corresponding image elements so that they are presented atadjacent or very close pixels. When they are viewed by a user, theyappear to be combined as a single unit.

Table 1 below is an illustration of a set of programmable instructionsfor combining corresponding data elements of two data sets of visuallydisplayable data by subtracting a weighted intensity value of imageelements of the modulated unattended data from the intensity value ofcorresponding image elements of the attended data. The weight factorassociated with the unattended data may be user selected.

TABLE I % simple model program for acceleration-sensitive imagemodulation to % preempt/treat motion-sickness while performing visualtasks such as reading, watching movies, typing, etc. on an %accelerating platform such as a bus or boat. This code only reflects l %frame for modeling purpose only. Real code would generate a constantly %changing display by combining the 2 constantly changing data-streams.watermark_image = imread(‘tun01.jpg’); % fixed image/pattern file usedfor motion-sensitive luminance modulation % of display and continuouslyrefreshed % based on motion-sensitive hardware data watermark_contrast =0.3; % user-defined setting based on subjective effectivenesspreferences % to minimize distraction while maximizing effectiveness %while ON task_image = imread(‘screenshot1.jpg’); % dynamic image sent todisplay related to task and constantly refreshed % based on runningprograms and user interfacing % (what monitor would look like withoutapp) accelerateX = 00; % current acceleration horizontal coordinatemodified by accelerometers/gyroscopes etc. and constantly refreshedaccelerateY = 00; % vertical . . . accelerateZ = 1; %perpendicular/depth . . . rotationtheta = 0; % rotations %Matlab-specific code for performing watermark motion functions. % Otherprograms use different syntax to accomplish same goal. se =translate(strel(l), [accelerateX accelerateY]); watermark_image =imdilate(watermark_image,se); % watermark_image =imresize(watermark_image,accelerateZ); % watermark_image =imcrop(watermark_image,size(task_image));image(task_image-(watermark_image/(l/watermark_contrast))); axis(‘off’);

When mathematically combining values associated with a first and secondelement from the respective data sets of attended and unattended data,the first element may be weighted relative to the second element by auser selectable weight factor (e.g., watermark_contrast in the examplein Table 1), also referred to as the combination weight factor. Thisallows the user to control how much the attended data should dominatethe display relative to the unattended data. A user may even select thecombination weight factor to allow the unattended data to dominate thepresentation.

The combining may occur before or after the image elements are mapped toa pixel array of the display device. If it is performed before, thevalues associated with the corresponding image elements of the attendedand unattended data are combined. If it is performed after, the valuesassociated with the pixels that are mapped to the corresponding imageelements of the attended and unattended data are combined.

It is envisioned that before mathematically combining values of elementsfrom the attended and unattended data, the attended data is modified sothat the combining will be more effective. This includes adjustingextreme values which arc at or close to a maximum or minimum thresholdvalue that the element can be set to. The extreme value is adjusted tobe less extreme, i.e., more moderate, so that the combining will notexceed the threshold value. This is because if the combining exceeds thethreshold value the combined value will have to be set to the thresholdvalue, and the effect of the combining will be partially lost. Forexample, when the combining includes mathematically combining theluminance values of elements of the two sets of data, each element inthe attended data that has an extreme luminance value, e.g., thatcorresponds to black (when combining includes adding the luminance valueof the corresponding unattended element) or white (when combiningincludes subtracting the luminance value of the corresponding unattendedelement), is shifted to more moderate level, e.g., towards the grayrange. This way, the adding or subtracting will be more effective ornoticeable since the combining will have a greater effect on the extremevalues.

In another embodiment, the combination includes using similar locations,but not the same locations for displaying corresponding pixels of theattended and the unattended data. For example, individual pixels or setsof pixels of the unattended data could be displayed at a location nearor adjacent to the location that is associated with those pixels. Theaffect is to weave the display of the unattended data into the displayof the attended data. In this way both data sets are displayedsubstantially simultaneously. If the display resolution is high enough,each display might be comprised of pixels close enough to each other toenable the visual system to selectively attend to and selectivelyconstruct the attended pattern, while his/her subconscious successfullyconstructs the unattended pattern sensitive to the displaysaccelerations. As such, the two displays only occupy illusorily and/orapproximately same locations though they actually are displayed indifferent pixels on the device.

The MSM system 100 is adapted and configured so that the user may selecta variety of parameters. The user may select which data presentationdevice 104 or medium of presentation to use (e.g., audio, video, ortactile). The user may select a pattern or series of patterns from thedatabase of unattended basic patterns 204. The user may also selectparameters associated with the modulation, such as which types of motionwill trigger modulation (e.g., rotational, angular, and/or linearaccelerations; for a boat, motion caused by waves only; accelerationsensitivity (1.sup.st derivative); jerk-sensitivity (2.sup.ndderivative, or beyond), etc.); how the unattended basic pattern ismodified based on the motion data (e.g., which characteristics of thepattern are shifted, whether the entire pattern or selected objects ofthe pattern are shifted, the size of the shift relative to the sensedmotion, whether the shift is steady or intermittent (e.g.,stroboscopic), or which dimensions in 2-dimensional data (e.g.,unattended display data) correspond to sensed motion in 3-dimensionalspace).

The user may further select how elements of the attended and unattendeddata sets will be combined (e.g., weighting one data set, selectingwhich mathematical operations and functions to use for the combining).Additionally, the weighting may further be user selected so that a firstset of elements are associated with a first weight and a second set ofelements are associated with a second weight. For example, unattendeddata pixels that are located around the periphery of the screen may beweighted higher than the pixels mat are not around the periphery so thatthe unattended data is more visible in the peripheral area. Such animplementation might be useful for users who find it more useful to havea less visible unattended pattern e.g. nearer to the center of thedisplay and/or who find it more effective to have a more visibleunattended pattern in the periphery.

In conclusion, the MSM system 100 provides for presenting unattendeddata to the user while the user is attending to a presentation attendeddata. When on a moving platform, the unattended data may be modulated inaccordance with sensed motion. In a simulator or virtual realityapplication, the unattended data may be modulated in accordance withsimulated motion. The unattended and attended data are presentedsubstantially simultaneously to the user. They may be combined orpresented independently and substantially simultaneously, but in eithercase, the unattended data does not obstruct or interfere with theattended data.

The MSM system 100 may be used for a variety of applications. Whenproviding for concurrent visual presentation of the unattended andattended data sets, the MSM system 100 can be used as tool fordiagnosing or researching cognitive/visual processing. Anotherapplication of the MSM system 100 includes using the concurrentpresentation of unattended and attended data for multitasking, where theunattended data does not require constant attention but requiresevent-monitoring.

In such a case, the user can attend to the attended data withoutattending to the concurrently presented unattended data. When the user'sattention is required for the unattended data, such as upon theoccurrence of an event, the user is presented with an alert, e.g., anaudio, visual or tactile alert, such as an auditory beep or a pop-upwindow. At that point, the user can select to change the presentation ofthe unattended data so that the user can attend to it. The user mayselect to discontinue the combining of the unattended and attended data,such as to view the unattended data without combining it with theattended data, e.g., in a separate window of a computer display.Alternatively, the user may select to change the combination weightfactor so that the unattended data is more perceivable.

The MSM system 100 may be used by advertisers to present advertisementsads concurrently with attended data. For example, an advertisement maybe presented as unattended data, e.g. logos, banner ads with or withouttext, and/or videos, that is combined with attended data, such as a webpage, text file, or video. In this fashion, the unattended data issensed and processed by user without the user being required to attendto it. The user does not have to divert attention from the attended dataor a related task or take any actions related to the unattended data.Since the unattended data can be presented in the same visual space asthe attended data, the advertisement is not invisible.

The present method of concurrently presenting the advertisement asunattended data combined with attended data provides advantages overcurrently available advertising methods used with display screens.Unlike banner ads, the advertisement does not exclusively occupy any ofthe finite real-estate of the display screen because the attended datacan still fill the entire display screen. Similarly, unlike commercialsand previews, the advertisement does not interrupt or delay the attendeddata, as it appears concurrently with the attended data. The unattendeddata can fill the entire display or just a region of the display. Thatregion can be user selectable.

FIG. 6 shows how the MSM system 100 can provide for advertising whileminimizing motion sickness of a user who is attending to data displayedon the display screen of a computing device. First and second screenshots 602 and 604 are shown displayed on a handheld computing device.The screen shots 602 and 604 show a combined display 606 in which anattended pattern and an unattended pattern are combined. The attendedpattern includes a video stream that the user is attending to, and theunattended pattern includes one or more logos 608. The unattendedpattern is modulated in accordance with input motion data and/or userselection. The modulation accounts for the shift in size of the logo 608in screen shot 604 relative to screen shot 602. In shot 604 the contrastof the logo 608 relative to the attended data is less than that of shot602. The logo 608 in shot 604 is therefore much less noticeable. Theuser may have selected to reduce the contrast in shot 608 to a levelthat was not distracting to the user but provided enough input forproviding motion sickness relief.

FIG. 6 may also represent a system and method for presenting advertisingto a user who is attending to data displayed on the display screen of acomputing device without distracting the user from the data that theuser wants to attend to. This embodiment need not include motiondetection or modulation of unattended data based on the motiondetection. The logo 608 is visible to the user. Particularly in shot604, the logo does not distract the user from the data that the user isattending to. The unattended data may be static. Alternatively, theunattended data and logo may be modulated in accordance with a selectedroutine or based on one or more selected inputs. The modulation mayincrease the user's awareness of the logo, even on a subconscious level,without distracting the user from the attended data. Unattendedadvertising data need not be still-frame logos but can also even appearas video ads which similarly modulate attended data such as text in adigital book. The video frames would also appear as modulation of theattended data and may or may not be sensitive to motion.

The user may increase or decrease the visibility of the unattended datarelative to the attended data at will, such as by adjusting thecombination weight factor. The user may choose to temporarily attend tothe unattended data while still viewing or monitoring the attended data.Accordingly, the user may switch between attending to the unattendeddata and the attended data at will. For example, the attended data mayinclude a movie or a televised ball game and the unattended data mayinclude an interactive video game. The user may wish to take a partialbreak from attending to the attended data while still viewing ormonitoring the attended data and temporarily adjust the combinationweight factor so that the unattended data is more visible than theattended data.

FIG. 7 shows a system and method for providing a business model fordistributing advertising via an unattended pattern that is combined withan attended pattern and displayed to a user. Advertisement system 700includes a web server 702, a plurality of user computing devices (userdevices) 704, and a plurality of vendor computing devices (vendordevices) 706. Exemplary user devices 704 labeled 1-5 and exemplaryvendor devices 706 labeled 1-3 are shown. Web server 702 communicateswith the user computing devices 704 and vendor computing devices 706 viaa network, such as the Internet or an intranet. Web server 702 is acomputing device that provides a web page that the computing devices 704and 706 can access for receiving and/or transmitting data to and fromthe web server 702. The computing devices 702, 704, and 706 each includea hardware processor for executing software programmable instructions, astorage device, communication software and hardware for communicatingwith other computing devices via the network, and hardware and softwarefor interfacing with a user or administrator. The computing devices 702,704, 706 may include a data stream combination module 208 and/or amotion function module 206 for execution by the associated processor.

The vendor devices 706 are each associated with a commercial vendor thatseeks to advertise product or service. The vendor devices 706 may eachhave an arrangement with web server 702 for facilitating advertising byincluding their respective logos in an unattended basic pattern that istransmitted by the web server 702 to one or more user devices 704. Thearrangement between the vendor devices 706 and web server 702 mayinclude a financial transaction or a form of bartering goods orservices. Alternatively, as shown by vendor device 3 and user device 5,the vendor device 706 may transmit an unattended basic pattern to theuser device 704 without using the web server 702.

The user devices 704 receive one or more unattended basic patterns formthe web server 702 or the vendor devices 706. This arrangement may alsoinclude an exchange of money, goods, or services. For example, a useroperating a user device 704 may agree to receive advertising in the formof an unattended basic pattern or unattended data including theunattended basic pattern in exchange for a discount on a product orservice. In another example, a product, such as a movie or an electronicbook, may be provided to the user device together with at least oneunattended basic pattern. The product may be combined with theunattended basic pattern and transmitted to the user device 704 ascombined data (see user device 2). The combining may include combiningcorresponding pixels of the unattended data and the product byperforming a mathematical function on a value associated with a propertyof both of the corresponding elements.

The unattended data may or may not be modulated. Modulation may be inresponse to detected motion, in accordance with a selected routine, orbased on one or more other selected inputs. Modulation may be performedby the web server 702 or by the user device 704. When modulation isperformed by the web server 702 it may be performed in real time. Theuser device 704 may transmit detected motion to the web server 702 whichmay modulate the unattended data and transmit the modulated unattendeddata to the user device 704 for display by the user device 704.

Combination of the unattended data (whether modulated or not) withattended data may be performed by the web server 702 or the user device704. The attended data may be provided by the web server 702 or may beaccessed by the user device 704 without involvement of the web server702.

With reference to FIGS. 8 and 9 showing another embodiment of thedisclosure, an audio signal integration system 800 is provided. Audiosignal integration system 800 includes a data integration device 804which receives two or more input audio data sets substantiallysimultaneously via audio input devices 802. Optionally, motion data asdescribed above with respect to MSM 100 may be received by the dataintegration device 804 via motion data input device 803. Dataintegration device 804 includes hardware processor 102 that executes oneor more of modules 206, 208, 210, and 212 for processing the input audiodata sets and/or combining them. Processor 102 executes data streamcombination module 208 for combining the two or more audio data sets sothat data from one data set does not interfere with data from the otherdata set. Data stream combination module 208 outputs combined audiodata.

The combining may include applying a mathematical function to a valueassociated with corresponding signals of each data set or interleavingsignals. Values may be associated with characteristics of the signalssuch as pitch, quality and loudness. Signals may correspond, forexample, in accordance with their position in an array of signals and/orin a series of signals.

The user may influence the combining via user input controls so that theuser can attend to a selected one of the input audio data sets. The usermay operate the user input controls to affect the combining andalternate between which input audio data set the user will attend to.Even when the output combined audio data is combined, such as byapplying a mathematical function or interleaving corresponding signalsfrom the two data sets, the combining is such that the data set selectedfrom the two or more data sets is not overpowered by the other dataset(s) and the user can recognize it and attend to it without attendingto the other unselected data set(s).

The two or MOM data sets may be audio signals (analog or digital)independently generated by respective audio data sources (not shown),such as a DVD player, a radio; a laptop computer; a television screen;or a handheld computing device, such as a smartphone, a personal digitalassistant (PDA), an MP3 player, or a cellular phone. Audio input devices802 couple between the audio data source and processor 102. Audio inputdevices 802 are illustrated as being external to data integration device804, but it is contemplated that one or more audio data sources andaudio input devices 802 may be integrated with the data integrationdevice 804 and disposed within data integration device's 804 housing.Examples of audio input devices 802 include surface-mounted connectorsor jacks such as TS, TRS, or TRRS connectors.

Motion function module 206 may modify at least one of the attended orunattended data with the motion data. The audio data sets may bemodulated in accordance with motion data and/or input other than motiondata, such as user input.

Data integration device 804 or audio output device 806 may includeassociated user input controls for allowing the user to adjustcharacteristics associated with the incoming data sets (e.g., volume,pitch, quality, etc.), parameters associated with the modulation,parameters for combining the data sets (e.g., selecting which data setthe user wants to attend to, selecting a weight factor, selecting whichmathematical operations and functions to use for the combining), and thecharacteristics associated with the output data.

Audio output device 806 is coupled to the data integration device 804for receiving the combined audio data, and may additionally providecontrol signals to the data integration device 804. The audio outputdevice 806 includes one or more speakers or microphones for outputtingthe combined audio data as sound. The audio output device 806 mayinclude a one or more speakers or microphones. For example the audiooutput device 806 may be configured as a binaural device, such as aheadset.

The data integration device 804 may operate with multiple users, eachuser having access to its own user input controls. Each user may selecttwo or more input data sets to be combined and/or modulated inaccordance with selections made via the user's user controls. The datastream combination module 208 outputs different combined audio dataassociated with the respective multiple users.

In an exemplary application of combining two audio data streams, a usermay be exercising while watching a television show together with otherviewers. The user may prefer to listen to music while listening to thetelevision show together with the other viewers. The music may beprovided as unattended data which is not attended to by the user on aconscious level, but is used for the unconscious task of keeping arhythm while exercising. At the same time, the user can attend to thetelevision program that is being presented as attended data. The userlistens to the combined data through a headset via which the user canlisten to only music, only the television show audio, or a combinationof the two.

The combination may be perceived as two substantially simultaneous soundstreams, such as with one of the sound streams presented at a lowervolume and less perceptible. The user can select which of the soundstreams the user wishes to attend to while still having the ability tosense and process the other sound stream without attending to it. Theuser can alternate at will between which sound stream the user wishes toattend to. Furthermore, the user can choose to modulate each of thesound streams in accordance with the user's needs or preferences. At thesame time, the other viewers may listen to the television audio viaanother speaker or individual headsets independently of the dataintegration device 804. Alternatively, these additional speakers may becoupled to an independent data integration device, or coupled with theuser's data integration device 804 for receiving an output combinationsignal that is modulated and/or combined in accordance with each user'sinput selections.

One of the sound streams may be modulated based on received motion data.For example, when the users are positioned on a moving platform, eachuser may select a different unattended basic pattern that is modulatedbased on the motion data and in accordance with each user's selectedparameters.

It will be appreciated that the above-disclosed and other features andfunctions, or alternatives thereof, may be desirably combined into manyother different systems or applications. Also that various presentlyunforeseen or unanticipated alternatives, modifications, variations orimprovements therein may be subsequently made by those skilled in theart which are also intended to be encompassed by the following claims.

What is claimed is:
 1. A stimulus-integrating apparatus for presentingsensory stimuli to a single sensory receptor of a user, the apparatuscomprising a medium, the medium storing machine-readable instructions,the instructions, when executed by a processor, instruct the processorto store in memory a single value, the value encoding a single stimuluspresentable to the single receptor, the stimulus including both attendeddata and unattended data, the attended data comprising information forconscious attention by the user, and the unattended data comprisinginformation for unconscious processing by the user, the unconsciousprocessing mediating a neurophysiological benefit for the user.
 2. Theapparatus of claim 1 wherein the neurophysiological benefit includes atleast one of: motion sickness prevention and motion-sicknessalleviation.
 3. The apparatus of claim 1 wherein the neurophysiologicalbenefit includes maintaining a rhythm while exercising.
 4. The apparatusof claim 1 wherein the stimulus includes an auditory stimulus.
 5. Theapparatus of claim 4 wherein the auditory stimulus is presented via aspeaker in electronic communication with a plurality of auditory dataprocessing devices, a first of the devices processing the unattendeddata and a second of the devices processing the attended data, theprocessing of the first device substantially simultaneous to theprocessing of the second device.
 6. The apparatus of claim 1 wherein thestimulus includes a visual stimulus.
 7. The apparatus of claim 1 whereinthe stimulus is included in an array of stimuli, each of the stimuliincluding both attended data and unattended data.
 8. The apparatus ofclaim 1 wherein the unattended data includes a plurality of oscillatingconcentric shapes presented on a visual display.
 9. The apparatus ofclaim 1 wherein the unattended data is coded to appear on a visualdisplay as: translucent with respect to the attended data; moving overthe attended data; and includes at least one of: a geometric shape; anda pattern.
 10. The apparatus of claim 1 wherein the unattended data isconfigured to be interpreted by a user as comprising no more than twocolors.
 11. The apparatus of claim 1 wherein the stimulus is coded forpresentation via a single pixel.
 12. The apparatus of claim 1 whereinthe unattended data and the attended data are coded to appear asoriginating from the same location.
 13. A stimulus integrating apparatuscomprising machine-readable instructions that when executed by aprocessor, instruct the processor to present, substantiallysimultaneously, via a single pixel of a display, a first set ofinformation for conscious attending by a user and a second set ofinformation for unconscious processing by the user, the first setcomprising an element of a first visual scene and the second setcomprising an element of a second visual scene, the first sceneunrelated to the second scene.
 14. The apparatus of claim 13 wherein thefirst set of information and the second set of information are bothcoded to be implicitly interpolated, by the visual system of the user,with data presented via pixels adjacent the single pixel, therebycreating substantially simultaneous presentation of both the first sceneand the second scene via each of the pixels.
 15. The apparatus of claim13 wherein the second scene includes advertising data.
 16. The apparatusof claim 15 wherein the advertising data is transmitted over a network.17. An electronic apparatus comprising: a stimulus presenter; and amedium including machine-readable instructions, the instructions forpresenting, via the presenter, to a single sensory receptor of a user, asensory stimulus substantially simultaneously including: attended dataincluding a first set of motion information for conscious attention bythe user; and unattended data including a second set of motioninformation for implicit processing by the user.
 18. The apparatus ofclaim 17 wherein the instructions, when executed by a processorcontrolling the presenter, instruct the processor to perform modulationsof both the unattended data and the attended data, the modulationspreventing: the unattended data from interfering with the attended data;and the attended data from interfering with the unattended data.
 19. Theapparatus of claim 17 wherein: the first set of motion informationincludes at least one of: a moving scene in at least one of a videogameand a movie; a scrolling page including text; and a cursor; and thesecond set of motion information includes a translucent moving patternconfigured to convey a sense of motion, the pattern comprising no morethan three colors, the moving including at least one of: oscillating;and translating in response to data transmitted by a motion sensingdevice, the device selected from: an accelerometer; a camera; agyroscope; a GPS device; and a WIFI network detector.
 20. The apparatusof claim 17 wherein the attended data comprises a first array of valuesand the unattended data comprises a second array of values and thepresenting includes subtracting the second set of values from the firstset of values.